{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameters tuning\n",
    "\n",
    "\n",
    "This tutorial will show you how to tune hyperparameters in CatBoost using built-in capabilities, [optuna](https://github.com/optuna/optuna), and [hyperopt](https://github.com/hyperopt/hyperopt) frameworks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip3 install catboost\n",
    "!pip3 install optuna\n",
    "!pip3 install hyperopt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "We are going to tune hyperparameters for a binary classsification task on the [UCI Adult Dataset](https://archive.ics.uci.edu/ml/datasets/Adult). We will predict a person's annual income - whether they make more than 50K or not. Let's build 2 sets of catboost pools(train, validation, test) for that, one consisting only of numerical features and the other of numerical and categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education-num</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516.0</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>50.0</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311.0</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>38.0</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646.0</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>53.0</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721.0</td>\n",
       "      <td>11th</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409.0</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    age         workclass    fnlwgt  education  education-num  \\\n",
       "0  39.0         State-gov   77516.0  Bachelors           13.0   \n",
       "1  50.0  Self-emp-not-inc   83311.0  Bachelors           13.0   \n",
       "2  38.0           Private  215646.0    HS-grad            9.0   \n",
       "3  53.0           Private  234721.0       11th            7.0   \n",
       "4  28.0           Private  338409.0  Bachelors           13.0   \n",
       "\n",
       "       marital-status         occupation   relationship   race     sex  \\\n",
       "0       Never-married       Adm-clerical  Not-in-family  White    Male   \n",
       "1  Married-civ-spouse    Exec-managerial        Husband  White    Male   \n",
       "2            Divorced  Handlers-cleaners  Not-in-family  White    Male   \n",
       "3  Married-civ-spouse  Handlers-cleaners        Husband  Black    Male   \n",
       "4  Married-civ-spouse     Prof-specialty           Wife  Black  Female   \n",
       "\n",
       "   capital-gain  capital-loss  hours-per-week native-country income  \n",
       "0        2174.0           0.0            40.0  United-States  <=50K  \n",
       "1           0.0           0.0            13.0  United-States  <=50K  \n",
       "2           0.0           0.0            40.0  United-States  <=50K  \n",
       "3           0.0           0.0            40.0  United-States  <=50K  \n",
       "4           0.0           0.0            40.0           Cuba  <=50K  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import catboost\n",
    "from catboost.datasets import adult\n",
    "from catboost.utils import eval_metric\n",
    "\n",
    "adult_train, adult_test = adult()\n",
    "adult_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "numeric_features = ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']\n",
    "categorical_features = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']\n",
    "\n",
    "adult_train['income'] = adult_train['income'].map({'<=50K': 0., '>50K': 1.})\n",
    "adult_test['income'] = adult_test['income'].map({'<=50K': 0., '>50K': 1.})\n",
    "for c in categorical_features:\n",
    "    adult_train[c].fillna('nan', inplace=True)\n",
    "    adult_test[c].fillna('nan', inplace=True)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(adult_train[numeric_features + categorical_features], adult_train['income'], test_size=0.2, random_state=42)\n",
    "\n",
    "\n",
    "numeric_train_pool = catboost.Pool(X_train[numeric_features], y_train)\n",
    "numeric_val_pool = catboost.Pool(X_test[numeric_features], y_test)\n",
    "numeric_test_pool = catboost.Pool(adult_test[numeric_features], adult_test['income'])\n",
    "\n",
    "\n",
    "cat_train_pool = catboost.Pool(X_train, y_train, cat_features=categorical_features)\n",
    "cat_val_pool = catboost.Pool(X_test, y_test, cat_features=categorical_features)\n",
    "cat_test_pool = catboost.Pool(adult_test, adult_test['income'], cat_features=categorical_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function will calculate the quality of found hyperparameters on a test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_test_quality(train_pool=numeric_train_pool, val_pool=numeric_val_pool, test_pool=numeric_test_pool, **kwargs):\n",
    "    model = catboost.CatBoostClassifier(**kwargs, random_seed=42)\n",
    "    model.fit(train_pool, verbose=0, eval_set=val_pool)\n",
    "    y_pred = model.predict_proba(test_pool)\n",
    "    return eval_metric(test_pool.get_label(), y_pred[:, 1], 'AUC')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a model with default parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.8744062991309358]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_test_quality()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will optimize the folowing hyperparameters using numeric features only:\n",
    "- **learning_rate** - used for reducing the gradient step\n",
    "- **depth** - depth of the trees\n",
    "- **l2_leaf_reg** - coefficient at the L2 regularization term of the cost function\n",
    "- **boosting_type** - boosting scheme. Possible values: Ordered, Plain\n",
    "\n",
    "There are 2 built-in methods of hyperparameters tuning in CatBoost: grid search and random search. Grid search is simply an exhaustive searching through a manually specified subset of the hyperparameter space. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "bestTest = 0.3475488771\n",
      "bestIteration = 999\n",
      "\n",
      "0:\tloss: 0.3475489\tbest: 0.3475489 (0)\ttotal: 9.18s\tremaining: 5m 21s\n",
      "\n",
      "bestTest = 0.3468175664\n",
      "bestIteration = 594\n",
      "\n",
      "\n",
      "bestTest = 0.3481154366\n",
      "bestIteration = 999\n",
      "\n",
      "\n",
      "bestTest = 0.3454653087\n",
      "bestIteration = 975\n",
      "\n",
      "\n",
      "bestTest = 0.3488477618\n",
      "bestIteration = 991\n",
      "\n",
      "\n",
      "bestTest = 0.34700929\n",
      "bestIteration = 954\n",
      "\n",
      "\n",
      "bestTest = 0.3469102947\n",
      "bestIteration = 575\n",
      "\n",
      "\n",
      "bestTest = 0.3462598759\n",
      "bestIteration = 660\n",
      "\n",
      "\n",
      "bestTest = 0.3475839061\n",
      "bestIteration = 774\n",
      "\n",
      "\n",
      "bestTest = 0.3469096484\n",
      "bestIteration = 456\n",
      "\n",
      "\n",
      "bestTest = 0.3469822493\n",
      "bestIteration = 989\n",
      "\n",
      "\n",
      "bestTest = 0.3469540474\n",
      "bestIteration = 769\n",
      "\n",
      "\n",
      "bestTest = 0.3459891057\n",
      "bestIteration = 863\n",
      "\n",
      "\n",
      "bestTest = 0.3465270739\n",
      "bestIteration = 453\n",
      "\n",
      "\n",
      "bestTest = 0.3468671756\n",
      "bestIteration = 962\n",
      "\n",
      "\n",
      "bestTest = 0.347612502\n",
      "bestIteration = 323\n",
      "\n",
      "\n",
      "bestTest = 0.347555923\n",
      "bestIteration = 900\n",
      "\n",
      "\n",
      "bestTest = 0.3476361304\n",
      "bestIteration = 410\n",
      "\n",
      "\n",
      "bestTest = 0.3498501695\n",
      "bestIteration = 994\n",
      "\n",
      "\n",
      "bestTest = 0.3486111677\n",
      "bestIteration = 749\n",
      "\n",
      "\n",
      "bestTest = 0.3504785069\n",
      "bestIteration = 949\n",
      "\n",
      "20:\tloss: 0.3504785\tbest: 0.3454653 (3)\ttotal: 7m 10s\tremaining: 5m 7s\n",
      "\n",
      "bestTest = 0.3483364437\n",
      "bestIteration = 953\n",
      "\n",
      "\n",
      "bestTest = 0.3504618373\n",
      "bestIteration = 999\n",
      "\n",
      "\n",
      "bestTest = 0.3489928514\n",
      "bestIteration = 936\n",
      "\n",
      "\n",
      "bestTest = 0.3497296305\n",
      "bestIteration = 588\n",
      "\n",
      "\n",
      "bestTest = 0.3499512869\n",
      "bestIteration = 403\n",
      "\n",
      "\n",
      "bestTest = 0.3497524151\n",
      "bestIteration = 876\n",
      "\n",
      "\n",
      "bestTest = 0.3510245542\n",
      "bestIteration = 294\n",
      "\n",
      "\n",
      "bestTest = 0.3498795073\n",
      "bestIteration = 556\n",
      "\n",
      "\n",
      "bestTest = 0.3500383932\n",
      "bestIteration = 271\n",
      "\n",
      "\n",
      "bestTest = 0.3532776034\n",
      "bestIteration = 529\n",
      "\n",
      "\n",
      "bestTest = 0.3534400804\n",
      "bestIteration = 223\n",
      "\n",
      "\n",
      "bestTest = 0.3532153616\n",
      "bestIteration = 519\n",
      "\n",
      "\n",
      "bestTest = 0.3527127915\n",
      "bestIteration = 214\n",
      "\n",
      "\n",
      "bestTest = 0.3527405372\n",
      "bestIteration = 498\n",
      "\n",
      "\n",
      "bestTest = 0.353780516\n",
      "bestIteration = 267\n",
      "\n",
      "35:\tloss: 0.3537805\tbest: 0.3454653 (3)\ttotal: 9m 20s\tremaining: 0us\n",
      "Estimating final quality...\n"
     ]
    }
   ],
   "source": [
    "grid = {\n",
    "    'learning_rate': [0.03, 0.06],\n",
    "    'depth':[3, 6, 9],\n",
    "    'l2_leaf_reg': [2, 3, 4],\n",
    "    'boosting_type': ['Ordered', 'Plain']\n",
    "}\n",
    "\n",
    "grid_search_model = catboost.CatBoostClassifier(iterations=1000, random_seed=42)\n",
    "grid_search_result = grid_search_model.grid_search(grid, numeric_train_pool, verbose=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.8751648039383577],\n",
       " {'depth': 3,\n",
       "  'l2_leaf_reg': 3,\n",
       "  'learning_rate': 0.06,\n",
       "  'boosting_type': 'Ordered'})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_test_quality(**grid_search_result['params']), grid_search_result['params'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random search replaces the exhaustive enumeration of all combinations by selecting them randomly. In random search, we define distributions for hyperparameters instead of specific values. You can also pass a list of values - it will be sampled uniformly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "bestTest = 0.3570621798\n",
      "bestIteration = 994\n",
      "\n",
      "0:\tloss: 0.3570622\tbest: 0.3570622 (0)\ttotal: 9.38s\tremaining: 2m 58s\n",
      "\n",
      "bestTest = 0.3585051894\n",
      "bestIteration = 576\n",
      "\n",
      "\n",
      "bestTest = 0.354942476\n",
      "bestIteration = 398\n",
      "\n",
      "\n",
      "bestTest = 0.3536843023\n",
      "bestIteration = 995\n",
      "\n",
      "\n",
      "bestTest = 0.3549438598\n",
      "bestIteration = 445\n",
      "\n",
      "\n",
      "bestTest = 0.3548070331\n",
      "bestIteration = 998\n",
      "\n",
      "5:\tloss: 0.3548070\tbest: 0.3536843 (3)\ttotal: 1m 1s\tremaining: 2m 22s\n",
      "\n",
      "bestTest = 0.3541591107\n",
      "bestIteration = 986\n",
      "\n",
      "\n",
      "bestTest = 0.3531888405\n",
      "bestIteration = 881\n",
      "\n",
      "\n",
      "bestTest = 0.3538994015\n",
      "bestIteration = 980\n",
      "\n",
      "\n",
      "bestTest = 0.3525667359\n",
      "bestIteration = 999\n",
      "\n",
      "\n",
      "bestTest = 0.3526242018\n",
      "bestIteration = 377\n",
      "\n",
      "10:\tloss: 0.3526242\tbest: 0.3525667 (9)\ttotal: 2m 30s\tremaining: 2m 3s\n",
      "\n",
      "bestTest = 0.354935685\n",
      "bestIteration = 835\n",
      "\n",
      "\n",
      "bestTest = 0.3561806368\n",
      "bestIteration = 878\n",
      "\n",
      "\n",
      "bestTest = 0.3616929754\n",
      "bestIteration = 997\n",
      "\n",
      "\n",
      "bestTest = 0.356413664\n",
      "bestIteration = 321\n",
      "\n",
      "\n",
      "bestTest = 0.3570983676\n",
      "bestIteration = 369\n",
      "\n",
      "15:\tloss: 0.3570984\tbest: 0.3525667 (9)\ttotal: 3m 3s\tremaining: 46s\n",
      "\n",
      "bestTest = 0.3567442074\n",
      "bestIteration = 257\n",
      "\n",
      "\n",
      "bestTest = 0.3567978468\n",
      "bestIteration = 654\n",
      "\n",
      "\n",
      "bestTest = 0.3578015136\n",
      "bestIteration = 475\n",
      "\n",
      "\n",
      "bestTest = 0.3576902685\n",
      "bestIteration = 184\n",
      "\n",
      "19:\tloss: 0.3576903\tbest: 0.3525667 (9)\ttotal: 3m 34s\tremaining: 0us\n",
      "Estimating final quality...\n"
     ]
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "params_distribution = {\n",
    "    'learning_rate': stats.uniform(0.01, 0.1),\n",
    "    'depth': list(range(3, 10)),\n",
    "    'l2_leaf_reg': stats.uniform(1, 10),\n",
    "    'boosting_type': ['Ordered', 'Plain'],\n",
    "}\n",
    "\n",
    "random_search_model = catboost.CatBoostClassifier(random_seed=42)\n",
    "random_search_result = random_search_model.randomized_search(\n",
    "    params_distribution, \n",
    "    numeric_train_pool, \n",
    "    n_iter=20, \n",
    "    verbose=5, \n",
    "    partition_random_seed=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.8767458281765127],\n",
       " {'depth': 8,\n",
       "  'learning_rate': 0.026233677654785993,\n",
       "  'l2_leaf_reg': 6.629482011941916,\n",
       "  'boosting_type': 'Ordered'})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_test_quality(**random_search_result['params']), random_search_result['params']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayesian optimization\n",
    "\n",
    "Random and grid search pay no attention to past results when searhing the best hyperparametes. Bayesian optimization, in contrast to random or grid search, keeps track of past evaluation results which it uses to form a probabilistic model mapping hyperparameters to a probability of a score on the objective function. There're a number of libraries that can do it. In this tutorial we will try 2 of them:  [optuna](https://github.com/optuna/optuna) and [hyperopt](https://github.com/hyperopt/hyperopt)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with optuna: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-05-02 16:17:55,634]\u001b[0m A new study created in memory with name: no-name-f2f48a39-17c5-4111-8df5-b5d050c48f1f\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:18:48,836]\u001b[0m Trial 0 finished with value: 0.8836909164770923 and parameters: {'learning_rate': 0.07268222670380756, 'depth': 9, 'l2_leaf_reg': 4.856238335681431, 'boosting_type': 'Ordered'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:18:56,100]\u001b[0m Trial 1 finished with value: 0.8808166455904405 and parameters: {'learning_rate': 0.05961832921746022, 'depth': 6, 'l2_leaf_reg': 5.420070400893375, 'boosting_type': 'Plain'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:19:02,431]\u001b[0m Trial 2 finished with value: 0.8817266285087795 and parameters: {'learning_rate': 0.0982687778546154, 'depth': 3, 'l2_leaf_reg': 6.217248673203491, 'boosting_type': 'Plain'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:19:15,288]\u001b[0m Trial 3 finished with value: 0.880400230194122 and parameters: {'learning_rate': 0.04529057663747355, 'depth': 9, 'l2_leaf_reg': 6.64585308403855, 'boosting_type': 'Plain'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:19:49,089]\u001b[0m Trial 4 finished with value: 0.8827664433848943 and parameters: {'learning_rate': 0.0494715020211662, 'depth': 8, 'l2_leaf_reg': 7.252796594667421, 'boosting_type': 'Ordered'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:21:21,745]\u001b[0m Trial 5 finished with value: 0.8830979141620132 and parameters: {'learning_rate': 0.07641958651588322, 'depth': 10, 'l2_leaf_reg': 4.962314602576686, 'boosting_type': 'Ordered'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:21:30,786]\u001b[0m Trial 6 finished with value: 0.8802874644411134 and parameters: {'learning_rate': 0.05783962364576546, 'depth': 7, 'l2_leaf_reg': 4.8507714175403756, 'boosting_type': 'Plain'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:21:45,098]\u001b[0m Trial 7 finished with value: 0.8822996665843196 and parameters: {'learning_rate': 0.08644886146700106, 'depth': 4, 'l2_leaf_reg': 7.662667516613093, 'boosting_type': 'Ordered'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:21:59,422]\u001b[0m Trial 8 finished with value: 0.8823247184849022 and parameters: {'learning_rate': 0.07501990443131994, 'depth': 4, 'l2_leaf_reg': 6.892491767228378, 'boosting_type': 'Ordered'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:22:05,732]\u001b[0m Trial 9 finished with value: 0.8810902200728965 and parameters: {'learning_rate': 0.030543690779106007, 'depth': 4, 'l2_leaf_reg': 9.891957578650006, 'boosting_type': 'Plain'}. Best is trial 0 with value: 0.8836909164770923.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:23:33,010]\u001b[0m Trial 10 finished with value: 0.8837636893502503 and parameters: {'learning_rate': 0.018811391183601193, 'depth': 10, 'l2_leaf_reg': 1.7615014321430658, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:24:58,357]\u001b[0m Trial 11 finished with value: 0.8827381714456762 and parameters: {'learning_rate': 0.012562144940540667, 'depth': 10, 'l2_leaf_reg': 1.6761725561945022, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:25:51,104]\u001b[0m Trial 12 finished with value: 0.883276626306273 and parameters: {'learning_rate': 0.015529600785637365, 'depth': 9, 'l2_leaf_reg': 1.3624523770968953, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:27:40,719]\u001b[0m Trial 13 finished with value: 0.8828463647438227 and parameters: {'learning_rate': 0.029350021633666297, 'depth': 10, 'l2_leaf_reg': 3.0522194118862744, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:28:25,583]\u001b[0m Trial 14 finished with value: 0.8829692414181463 and parameters: {'learning_rate': 0.06998781510353098, 'depth': 8, 'l2_leaf_reg': 2.717151821789632, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:28:40,712]\u001b[0m Trial 15 finished with value: 0.8822751298899185 and parameters: {'learning_rate': 0.09524510715640018, 'depth': 6, 'l2_leaf_reg': 3.3381716609815077, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:29:42,406]\u001b[0m Trial 16 finished with value: 0.882301985012137 and parameters: {'learning_rate': 0.04034390671992559, 'depth': 9, 'l2_leaf_reg': 8.452140769210578, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:30:16,386]\u001b[0m Trial 17 finished with value: 0.8825818063695455 and parameters: {'learning_rate': 0.06524881154390634, 'depth': 8, 'l2_leaf_reg': 3.858668765263367, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:30:41,149]\u001b[0m Trial 18 finished with value: 0.8828490695762764 and parameters: {'learning_rate': 0.08575881624121307, 'depth': 7, 'l2_leaf_reg': 1.0686090904107663, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:32:13,444]\u001b[0m Trial 19 finished with value: 0.8835756390939481 and parameters: {'learning_rate': 0.02279702335681278, 'depth': 10, 'l2_leaf_reg': 2.1113733158619685, 'boosting_type': 'Ordered'}. Best is trial 10 with value: 0.8837636893502503.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import optuna\n",
    "from optuna.samplers import TPESampler\n",
    "from catboost.utils import eval_metric\n",
    "\n",
    "\n",
    "def objective(trial):\n",
    "    params = {\n",
    "        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.1),\n",
    "        'depth': trial.suggest_int('depth', 3, 10),\n",
    "        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1, 10),\n",
    "        'boosting_type': trial.suggest_categorical('boosting_type', ['Ordered', 'Plain']),\n",
    "    }\n",
    "\n",
    "    model = catboost.CatBoostClassifier(**params, random_seed=42)\n",
    "    model.fit(numeric_train_pool, verbose=0, eval_set=numeric_val_pool)\n",
    "    y_pred = model.predict_proba(numeric_val_pool)\n",
    "    return eval_metric(numeric_val_pool.get_label(), y_pred[:, 1], 'AUC')\n",
    "\n",
    "sampler = TPESampler(seed=123)\n",
    "study = optuna.create_study(direction='maximize', sampler=sampler)\n",
    "study.optimize(objective, n_trials=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the logs above, Ordered boosting type is used more often than Plain. It shows that optuna learnt that Ordered boosting type scores better than Plain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.8768748819916609],\n",
       " {'learning_rate': 0.018811391183601193,\n",
       "  'depth': 10,\n",
       "  'l2_leaf_reg': 1.7615014321430658,\n",
       "  'boosting_type': 'Ordered'})"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_test_quality(**study.best_params), study.best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'boosting_type': 'Plain', 'depth': 8, 'l2_leaf_reg': 6.3139710353627025, 'learning_rate': 0.07552270821291171}\n",
      "{'boosting_type': 'Ordered', 'depth': 6, 'l2_leaf_reg': 4.308614773826791, 'learning_rate': 0.08728243927833365}\n",
      "{'boosting_type': 'Ordered', 'depth': 5, 'l2_leaf_reg': 7.867792270156927, 'learning_rate': 0.09932662966595064}\n",
      "{'boosting_type': 'Ordered', 'depth': 3, 'l2_leaf_reg': 7.303807275407774, 'learning_rate': 0.0706129528849592}\n",
      "{'boosting_type': 'Plain', 'depth': 6, 'l2_leaf_reg': 4.1869403239777645, 'learning_rate': 0.0606353426143548}\n",
      "{'boosting_type': 'Ordered', 'depth': 7, 'l2_leaf_reg': 5.079708066414372, 'learning_rate': 0.039778097292241806}\n",
      "{'boosting_type': 'Ordered', 'depth': 8, 'l2_leaf_reg': 6.2552942709930175, 'learning_rate': 0.047109386241537723}\n",
      "{'boosting_type': 'Ordered', 'depth': 6, 'l2_leaf_reg': 6.468768565626519, 'learning_rate': 0.07822320859088815}\n",
      "{'boosting_type': 'Ordered', 'depth': 3, 'l2_leaf_reg': 2.6549804340741274, 'learning_rate': 0.07912181128582386}\n",
      "{'boosting_type': 'Ordered', 'depth': 6, 'l2_leaf_reg': 6.60185754074968, 'learning_rate': 0.07546269862838803}\n",
      "{'boosting_type': 'Ordered', 'depth': 3, 'l2_leaf_reg': 9.910884566094966, 'learning_rate': 0.012407314917274589}\n",
      "{'boosting_type': 'Plain', 'depth': 9, 'l2_leaf_reg': 6.467623778803967, 'learning_rate': 0.010459679258125895}\n",
      "{'boosting_type': 'Ordered', 'depth': 9, 'l2_leaf_reg': 8.434204337800717, 'learning_rate': 0.0176989864529083}\n",
      "{'boosting_type': 'Plain', 'depth': 3, 'l2_leaf_reg': 2.483728741507214, 'learning_rate': 0.038911805635082376}\n",
      "{'boosting_type': 'Plain', 'depth': 7, 'l2_leaf_reg': 9.33453069947452, 'learning_rate': 0.08878741925407074}\n",
      "{'boosting_type': 'Ordered', 'depth': 4, 'l2_leaf_reg': 3.4788949899842665, 'learning_rate': 0.0736485301842669}\n",
      "{'boosting_type': 'Plain', 'depth': 6, 'l2_leaf_reg': 8.201580476448104, 'learning_rate': 0.07243380483946275}\n",
      "{'boosting_type': 'Ordered', 'depth': 4, 'l2_leaf_reg': 1.1394908616027137, 'learning_rate': 0.0415235054229544}\n",
      "{'boosting_type': 'Ordered', 'depth': 7, 'l2_leaf_reg': 8.481950903245354, 'learning_rate': 0.03051852105735192}\n",
      "{'boosting_type': 'Plain', 'depth': 9, 'l2_leaf_reg': 9.533316339225617, 'learning_rate': 0.04269145107403636}\n",
      "100%|██████████| 20/20 [05:33<00:00, 16.67s/trial, best loss: -0.8831551664489491]\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe\n",
    "import numpy as np\n",
    "\n",
    "def hyperopt_objective(params):\n",
    "    print(params)\n",
    "    model = catboost.CatBoostClassifier(**params, random_seed=42)\n",
    "    model.fit(numeric_train_pool, verbose=0, eval_set=numeric_val_pool)\n",
    "    y_pred = model.predict_proba(numeric_val_pool)\n",
    "    return -eval_metric(numeric_val_pool.get_label(), y_pred[:, 1], 'AUC')[0]\n",
    "\n",
    "space = {\n",
    "    'learning_rate': hp.uniform('learning_rate', 0.01, 0.1),\n",
    "    'depth': hp.randint('depth', 3, 10),\n",
    "    'l2_leaf_reg': hp.uniform('l2_leaf_reg', 1, 10),\n",
    "    'boosting_type': hp.choice('boosting_type', ['Ordered', 'Plain']),\n",
    "}\n",
    "\n",
    "best = fmin(hyperopt_objective,\n",
    "    space=space,\n",
    "    algo=tpe.suggest,\n",
    "    max_evals=20,\n",
    "    rstate=np.random.default_rng(seed=123))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.8758792836635058],\n",
       " {'boosting_type': 'Ordered',\n",
       "  'depth': 6,\n",
       "  'l2_leaf_reg': 4.308614773826791,\n",
       "  'learning_rate': 0.08728243927833365})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params = best.copy()\n",
    "best_params['boosting_type'] = 'Plain' if best['boosting_type'] == 1 else 'Ordered'\n",
    "calc_test_quality(**best_params), best_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical features\n",
    "\n",
    "Let's apply bayesian optimization approaches to a dataset with categorical features. We will also optimize **max_ctr_complexity** parameter - the maximum number of features that can be combined. Each resulting combination consists of one or more categorical features and can optionally contain binary features in the following form: “numeric feature > value”. But first let's train a model with default parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9273316513681858]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_test_quality(train_pool=cat_train_pool,val_pool=cat_val_pool,test_pool=cat_test_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2021-05-02 16:39:39,439]\u001b[0m A new study created in memory with name: no-name-26b54ca2-7882-4537-8ea3-bc0198fd2dbf\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:41:03,976]\u001b[0m Trial 0 finished with value: 0.9337410717988759 and parameters: {'learning_rate': 0.07268222670380756, 'depth': 9, 'l2_leaf_reg': 4.856238335681431, 'boosting_type': 'Ordered', 'max_ctr_complexity': 6}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:41:24,110]\u001b[0m Trial 1 finished with value: 0.9328366273469896 and parameters: {'learning_rate': 0.07472352791392958, 'depth': 5, 'l2_leaf_reg': 4.807958141120149, 'boosting_type': 'Ordered', 'max_ctr_complexity': 1}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:42:19,225]\u001b[0m Trial 2 finished with value: 0.9331029245421298 and parameters: {'learning_rate': 0.0716346764726377, 'depth': 9, 'l2_leaf_reg': 2.2595568636734606, 'boosting_type': 'Ordered', 'max_ctr_complexity': 0}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:42:44,026]\u001b[0m Trial 3 finished with value: 0.9331885775698291 and parameters: {'learning_rate': 0.07561447366456375, 'depth': 6, 'l2_leaf_reg': 3.202833492585279, 'boosting_type': 'Ordered', 'max_ctr_complexity': 0}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:43:07,266]\u001b[0m Trial 4 finished with value: 0.9326259724194674 and parameters: {'learning_rate': 0.06345121597066214, 'depth': 4, 'l2_leaf_reg': 2.6424255740815, 'boosting_type': 'Ordered', 'max_ctr_complexity': 4}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:43:21,899]\u001b[0m Trial 5 finished with value: 0.930844904649504 and parameters: {'learning_rate': 0.05783962364576546, 'depth': 7, 'l2_leaf_reg': 4.8507714175403756, 'boosting_type': 'Plain', 'max_ctr_complexity': 0}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:43:57,477]\u001b[0m Trial 6 finished with value: 0.9335577227989812 and parameters: {'learning_rate': 0.054287629911310815, 'depth': 6, 'l2_leaf_reg': 6.499211596098246, 'boosting_type': 'Ordered', 'max_ctr_complexity': 6}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:44:51,792]\u001b[0m Trial 7 finished with value: 0.9332833755072527 and parameters: {'learning_rate': 0.03906630224678604, 'depth': 9, 'l2_leaf_reg': 4.364212841200598, 'boosting_type': 'Ordered', 'max_ctr_complexity': 1}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:45:21,556]\u001b[0m Trial 8 finished with value: 0.9327990172957291 and parameters: {'learning_rate': 0.09891957578650006, 'depth': 6, 'l2_leaf_reg': 1.8289444595056765, 'boosting_type': 'Ordered', 'max_ctr_complexity': 2}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:46:34,971]\u001b[0m Trial 9 finished with value: 0.933408699410939 and parameters: {'learning_rate': 0.025651586461303076, 'depth': 9, 'l2_leaf_reg': 4.8324726126624515, 'boosting_type': 'Ordered', 'max_ctr_complexity': 4}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:47:23,766]\u001b[0m Trial 10 finished with value: 0.9294483738933693 and parameters: {'learning_rate': 0.09859192913304339, 'depth': 10, 'l2_leaf_reg': 9.591188896724713, 'boosting_type': 'Plain', 'max_ctr_complexity': 8}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:48:06,676]\u001b[0m Trial 11 finished with value: 0.9334483058861534 and parameters: {'learning_rate': 0.04322110526307429, 'depth': 7, 'l2_leaf_reg': 7.287628205892902, 'boosting_type': 'Ordered', 'max_ctr_complexity': 7}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:48:18,114]\u001b[0m Trial 12 finished with value: 0.9301199451511499 and parameters: {'learning_rate': 0.08618486264545978, 'depth': 3, 'l2_leaf_reg': 7.191163808878922, 'boosting_type': 'Plain', 'max_ctr_complexity': 6}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:49:14,540]\u001b[0m Trial 13 finished with value: 0.9332596116221241 and parameters: {'learning_rate': 0.04712000307225438, 'depth': 8, 'l2_leaf_reg': 6.763735509469435, 'boosting_type': 'Ordered', 'max_ctr_complexity': 6}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:49:41,456]\u001b[0m Trial 14 finished with value: 0.9306575628017015 and parameters: {'learning_rate': 0.016878692659184118, 'depth': 5, 'l2_leaf_reg': 9.197156646092772, 'boosting_type': 'Ordered', 'max_ctr_complexity': 5}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:50:40,648]\u001b[0m Trial 15 finished with value: 0.9333047565637912 and parameters: {'learning_rate': 0.08645689133705964, 'depth': 8, 'l2_leaf_reg': 6.1411841180059294, 'boosting_type': 'Ordered', 'max_ctr_complexity': 8}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:52:39,987]\u001b[0m Trial 16 finished with value: 0.9332167851082744 and parameters: {'learning_rate': 0.031005295194945375, 'depth': 10, 'l2_leaf_reg': 3.559233625616281, 'boosting_type': 'Ordered', 'max_ctr_complexity': 6}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:52:54,939]\u001b[0m Trial 17 finished with value: 0.9310954236553312 and parameters: {'learning_rate': 0.054922614429810494, 'depth': 5, 'l2_leaf_reg': 8.12743706885223, 'boosting_type': 'Plain', 'max_ctr_complexity': 7}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:53:53,895]\u001b[0m Trial 18 finished with value: 0.932792899222322 and parameters: {'learning_rate': 0.06508829676524042, 'depth': 8, 'l2_leaf_reg': 1.074204397099149, 'boosting_type': 'Ordered', 'max_ctr_complexity': 3}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n",
      "\u001b[32m[I 2021-05-02 16:54:14,605]\u001b[0m Trial 19 finished with value: 0.9319455138550534 and parameters: {'learning_rate': 0.08533130107354371, 'depth': 3, 'l2_leaf_reg': 6.210823895752981, 'boosting_type': 'Ordered', 'max_ctr_complexity': 5}. Best is trial 0 with value: 0.9337410717988759.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def objective(trial):\n",
    "    params = {\n",
    "        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.1),\n",
    "        'depth': trial.suggest_int('depth', 3, 10),\n",
    "        'l2_leaf_reg': trial.suggest_float('l2_leaf_reg', 1, 10),\n",
    "        'boosting_type': trial.suggest_categorical('boosting_type', ['Ordered', 'Plain']),\n",
    "        'max_ctr_complexity': trial.suggest_int('max_ctr_complexity', 0, 8)\n",
    "    }\n",
    "\n",
    "    model = catboost.CatBoostClassifier(**params, random_seed=42)\n",
    "    model.fit(cat_train_pool, verbose=0, eval_set=cat_val_pool)\n",
    "    y_pred = model.predict_proba(cat_val_pool)\n",
    "    return eval_metric(cat_val_pool.get_label(), y_pred[:, 1], 'AUC')\n",
    "\n",
    "sampler = TPESampler(seed=123)\n",
    "study = optuna.create_study(direction='maximize', sampler=sampler)\n",
    "study.optimize(objective, n_trials=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.9291097900449995],\n",
       " {'learning_rate': 0.07268222670380756,\n",
       "  'depth': 9,\n",
       "  'l2_leaf_reg': 4.856238335681431,\n",
       "  'boosting_type': 'Ordered',\n",
       "  'max_ctr_complexity': 6})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_test_quality(train_pool=cat_train_pool,\n",
    "                  val_pool=cat_val_pool,\n",
    "                  test_pool=cat_test_pool,\n",
    "                  **study.best_params), study.best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'boosting_type': 'Ordered', 'depth': 8, 'l2_leaf_reg': 9.02418957750765, 'learning_rate': 0.07874394992843053, 'max_ctr_complexity': 4}\n",
      "{'boosting_type': 'Ordered', 'depth': 3, 'l2_leaf_reg': 7.77026577135344, 'learning_rate': 0.08423590349470189, 'max_ctr_complexity': 0}\n",
      "{'boosting_type': 'Plain', 'depth': 9, 'l2_leaf_reg': 6.876425031781056, 'learning_rate': 0.043661293824597616, 'max_ctr_complexity': 7}\n",
      "{'boosting_type': 'Plain', 'depth': 7, 'l2_leaf_reg': 8.255101676728357, 'learning_rate': 0.09359026791928762, 'max_ctr_complexity': 6}\n",
      "{'boosting_type': 'Plain', 'depth': 6, 'l2_leaf_reg': 8.555879516541207, 'learning_rate': 0.012200281335930548, 'max_ctr_complexity': 7}\n",
      "{'boosting_type': 'Ordered', 'depth': 7, 'l2_leaf_reg': 4.577410159173021, 'learning_rate': 0.04272764683363647, 'max_ctr_complexity': 1}\n",
      "{'boosting_type': 'Ordered', 'depth': 5, 'l2_leaf_reg': 6.024099192696876, 'learning_rate': 0.08881365189368236, 'max_ctr_complexity': 7}\n",
      "{'boosting_type': 'Plain', 'depth': 7, 'l2_leaf_reg': 9.836210300493605, 'learning_rate': 0.0839390883944011, 'max_ctr_complexity': 5}\n",
      "{'boosting_type': 'Ordered', 'depth': 7, 'l2_leaf_reg': 1.7458831699694273, 'learning_rate': 0.03602803771607432, 'max_ctr_complexity': 7}\n",
      "{'boosting_type': 'Ordered', 'depth': 7, 'l2_leaf_reg': 6.699089582619849, 'learning_rate': 0.0986509585828973, 'max_ctr_complexity': 7}\n",
      "{'boosting_type': 'Plain', 'depth': 7, 'l2_leaf_reg': 3.9315321684767195, 'learning_rate': 0.028777104609525546, 'max_ctr_complexity': 0}\n",
      "{'boosting_type': 'Ordered', 'depth': 8, 'l2_leaf_reg': 7.149842992789483, 'learning_rate': 0.045634384339125734, 'max_ctr_complexity': 0}\n",
      "{'boosting_type': 'Ordered', 'depth': 3, 'l2_leaf_reg': 6.707356720205291, 'learning_rate': 0.07961213219420644, 'max_ctr_complexity': 2}\n",
      "{'boosting_type': 'Ordered', 'depth': 8, 'l2_leaf_reg': 7.653397594948659, 'learning_rate': 0.06471835281578665, 'max_ctr_complexity': 5}\n",
      "{'boosting_type': 'Plain', 'depth': 6, 'l2_leaf_reg': 5.429720790135397, 'learning_rate': 0.06726483049500963, 'max_ctr_complexity': 1}\n",
      "{'boosting_type': 'Ordered', 'depth': 5, 'l2_leaf_reg': 5.506271079099783, 'learning_rate': 0.09187172468594423, 'max_ctr_complexity': 0}\n",
      "{'boosting_type': 'Plain', 'depth': 7, 'l2_leaf_reg': 2.558021986753187, 'learning_rate': 0.0259480954856858, 'max_ctr_complexity': 3}\n",
      "{'boosting_type': 'Ordered', 'depth': 4, 'l2_leaf_reg': 5.521078771863403, 'learning_rate': 0.06646461830582402, 'max_ctr_complexity': 2}\n",
      "{'boosting_type': 'Plain', 'depth': 3, 'l2_leaf_reg': 1.2411392255204425, 'learning_rate': 0.04823793370142213, 'max_ctr_complexity': 0}\n",
      "{'boosting_type': 'Plain', 'depth': 6, 'l2_leaf_reg': 8.356139726566052, 'learning_rate': 0.048180319930115074, 'max_ctr_complexity': 2}\n",
      "100%|██████████| 20/20 [08:24<00:00, 25.23s/trial, best loss: -0.9336103382302823]\n"
     ]
    }
   ],
   "source": [
    "def hyperopt_objective(params):\n",
    "    print(params)\n",
    "    model = catboost.CatBoostClassifier(**params, random_seed=42)\n",
    "    model.fit(cat_train_pool, verbose=0, eval_set=cat_val_pool)\n",
    "    y_pred = model.predict_proba(cat_val_pool)\n",
    "    return -eval_metric(cat_val_pool.get_label(), y_pred[:, 1], 'AUC')[0]\n",
    "\n",
    "space = {\n",
    "    'learning_rate': hp.uniform('learning_rate', 0.01, 0.1),\n",
    "    'depth': hp.randint('depth', 3, 10),\n",
    "    'l2_leaf_reg': hp.uniform('l2_leaf_reg', 1, 10),\n",
    "    'boosting_type': hp.choice('boosting_type', ['Ordered', 'Plain']),\n",
    "    'max_ctr_complexity': hp.randint('max_ctr_complexity', 0, 8)\n",
    "}\n",
    "\n",
    "best = fmin(hyperopt_objective,\n",
    "    space=space,\n",
    "    algo=tpe.suggest,\n",
    "    max_evals=20,\n",
    "    rstate=np.random.default_rng(seed=123))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.9289912641941946],\n",
       " {'boosting_type': 'Ordered',\n",
       "  'depth': 7,\n",
       "  'l2_leaf_reg': 4.577410159173021,\n",
       "  'learning_rate': 0.04272764683363647,\n",
       "  'max_ctr_complexity': 1})"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params = best.copy()\n",
    "best_params['boosting_type'] = 'Plain' if best['boosting_type'] == 1 else 'Ordered'\n",
    "calc_test_quality(train_pool=cat_train_pool,\n",
    "                  val_pool=cat_val_pool,\n",
    "                  test_pool=cat_test_pool,\n",
    "                  **best_params), best_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "|Features|Opt algorithm|AUC score|\n",
    "|---|---|---|\n",
    "|Numeric|No opt|0.87440|\n",
    "|Numeric|Grid search|0.87516|\n",
    "|Numeric|Randomized search|0.87648|\n",
    "|Numeric|Optuna|**0.87687**|\n",
    "|Numeric|Hyperopt|0.87587|\n",
    "|Numeric & categorical|No opt|0.92733|\n",
    "|Numeric & categorical|Optuna|**0.92910**|\n",
    "|Numeric & categorical|Hyperopt|0.92899|"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
